% did you read the instructions
% did you have to ask someone on how to use the game
% if so what did you ask them?
% was the experience fun?
% Other comments?
\section{Evaluation Design}\label{sec_method}
To evaluate the project it required the creation of three experiments,  this section discusses the designs of the following experiments:
\begin{enumerate}
\item An evaluation of the usability and fun factor of the labelling game.
\item An investigation into the accessibility of  KAPTCHA for blind people.
\item To document an attack on KAPTCHA, in the hope that it proves resilient to the attack.
\end{enumerate}

\subsection{Evaluation of the game}\label{subsec_gameeval}
Although developed based on the design of implementation of games  previous implementations of labelling games (Gwap and Google labeler), the unique single player aspects of the game may not be as easy for a user to understand.  It may also be possible that the fun factor which the user experiences in game is due to the experience of playing with another user (see section \ref{subsect_riskfactors} for all potential project risks).  Participants were recruited from the pool of potential users, level three and four Computer Science students.  


The evaluation consisted of two main phases providing the user with an opportunity to play with the game then get them to fill out a questionnaire about how fun and usable the system was.  The users were directed to the website \url{http://bantayan.dcs.gla.ac.uk/testpage}, to read the instructions and play three iterations of the game. An iteration of the game is completed through successfully labeling of ten images. The participant was requested, once they had completed this task, to fill out a questionnaire. The evaluation took place in the level three and four computing science department labs.  The evaluation's purpose was not only used to determine the level of fun the user experiences while playing the game, but also to gauge how many people would play the game.  It was also hoped that the participants would inform their friends that they enjoyed playing the game. Which would hopefully increase the number of people who play the game.


\subsubsection{Software} - standard software was used in the experiment available in the level three and four labs.  In the level four lab Windows XP with Internet explorer and Firefox are available as well as Linux with the Firefox web browser.  The Level three labs are limited to Linux and Firefox.  The website required the user to have JavaScript enabled on their browser.  If JavaScript was not enabled then the website informed the user that this was the case and gave detailed instructions on how to enable it\ref{fig_javascriptinstructions}.

\begin{figure}[h]\label{fig_gameinstructions}
\centering
 \includegraphics[width=80mm]{instructions.eps}
\caption{Instructions for playing the game}
\end{figure}


\begin{figure}[h]\label{fig_javascriptinstructions}
\centering
 \includegraphics[width=80mm]{javascriptinstruct.eps}
\caption{Instructions for fixing inactive JavaScript}
\end{figure}


\subsection{Evaluation of the KAPTCHA}\label{subsec_captchaeval}
This is an outline of the Usability study to evaluate the level of usability the proposed CAPTCHA provides to blind people. The study will take place in the participant's office, all of which were located in the RNIB (Royal National Institute for the blind) in Glasgow UK.

Five blind users participated.  All were experienced in the use of computers, having been using computers without the aid of vision for many years.  The instructions as to how the system worked were read to the participants (see \ref{sec_apendix} for material used in the experiment). 


The participants were assigned a unique number which would allow for the distinction between the entered information, without having to store information which could be used to identify them personally.

The participants were informed that the amount of time taken to complete each KAPTCHA was being recorded and if they had any problems they should not hesitate to any questions.  As the office in which all of the participant's workstations were located, it was possible to be in the same room as all the participants, without directly monitoring their actions.  This was done in an attempt to minimise The Hawthorn Effect in which there is an increase on the performance of observed behaviour\cite{hawthorn}.  Each participant was tasked with the completion of ten KAPTCHAs. The participant was asked to fill out a questionnaire asking what they thought were of the system.  As the participants were blind an online questionnaire was provided, which allowed them to complete the task at their own pace and without aid.  The participant did not receive any feedback as to whether they were successful in the completion of a KAPTCHA. This was done to prevent rejection of the project by the Ethics committee board on the grounds that the system would cause the participant undue stress in the event of frequent incorrect answers.  The screen reader and web browser used in the experiment was to be of the participant's choosing and done at the participants desk. The evaluation  was conducted in the presence of a sighted RNIB worker whose task was collaborate and to ensure that the evaluation was performed to ethical guidelines.

\subsection{Evaluation of the Security of the  CAPTCHA}\label{subsec_attack}
To test the level of security, using the algorithm stated in section \ref{algor_attack}  the algorithm was implemented in Java and was performed on a AMD Athlon X2 dual-core processor with four Gigabytes of RAM.  This experiment was carried out in the university which is connected to JANET, which provides high speed internet to the United Kingdom's educational establishments.  The metrics taken are the amount of time it takes to complete an attack and whether the attack was successful or not.  Although the hardware used in the experiment is relatively new, it is available off of the shelf and access to high speed networks can be gained through purchasing a managed server.
